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What is Formative Research? 

In the design of educational computer software, for- 
mative research is responsible for assessing three 
fundamental issues: (1) comprehensibili^, (2) appeal, 
and (3) usefulness. Comprehensibility research inves- 
tigates both the clarity of the educational ideas con- 
tained within the software and the clarity of the inter- 
face that enables the user to interact vdth the program. 
Appeal research addresses the popularity of the basic 
activity among the target audience, and usefulness 
research evaluates the educational effectiveness of the 
product. Although these threeelementsconstitute the 
objectives of formative research, they tell us very li ttle 
about the actual role that the for-native researcher 
plays in the design process. 

The reality of the formative design process is far 
less neat. Since formative studies done in nonac- 
ademic settings are often conducted on a very small 
scale, with little time to reflect, revise, or reconsider, 
the researcher must invariably reach conclusions on 
the basis of incomplete evidence. Thus, a solid back- 
ground in developmental and educational psychol- 
ogy is necessary to place into a reliable context the 
impressions gathered from research. 

In many ways the formative researcher functions 
both as a troubleshooter for the designers and as an 
advocate for the target audience. She/he must be able 
to translate the concerns of the designers into quickly 
answerable research questions, and then translate the 
responses of the audience into recommendations for 
thedesigners. At times, she/he must also translate the 
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implementations of the designers into theoretically 
defensible pedagogical objectives for the educational 
establishment, and the suggestions of academic advi- 
sors into concrete guidelines for the production ^*aff. 
Throughout all ihis, the formative researcher must 
keep a clear view of how to make the best possible 
products for children. 

In this paper, I will illustrate the complexity of the 
tasks and skills required in formative research by pre- 
senting a case study of the design of an educational 
computer program, ''Maya Math,'' one of the class- 
room modules developed as part ofThe Second Voyage 
oftheMimi project at Bank Street College o^ Education. 
The Mirni classroom modules differ from the design 
premise of other educational software programs inso- 
far as they comprise both software and print materi- 
als. As a result, the software is not expected to carry 
the entire pedagogical burden. I will use the context of 
Maya Math to illustrate how the specific issues out- 
lined above came into play in the design of this learn- 
ing module. 

The Educational Context 

Before addressing the formative work we carried out, 
I will describe the context in which this software 
program was created and discuss some of the educa- 
tional objectives that formed the backdrop of the 
design process. The Second Voyage of the Mimi is a 
federally funded multimedia project in math and 
science education for upper elementary school stu- 
dents. The centerpiece of the Mimi is a 24-part televi- 
sion show that focuses on Maya archeology in the 
Yucatan Peninsula of Mexico. The story features a 
young boy and his grandfather, who become involved 
in an adventure that leads them into an extensive ex- 
ploration of Maya life — both past and present. 
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The print materials encourage the teachers and 
students to use an open-ended hypothesis building 
and cebting approach as ihey try to make sense of the 
Maya system of writing numbers. In the course of 
their investigations, students encounter basic mathe- 
matical concepts, such as numeration and the inven- 
tion of zero. Most important, Maya Mrith serves as a 
reintroduction to the concept of place value. 

The computer software features a "Maya Calcula- 
tor," which functions as a tool used to facilitate the 
process of understanding the Maya base twenty sys- 
tem and other bases as vv^ell (see Figure 2). 

The software also includes a Maya number game, 
"Glyph Trek," which is modeled on the adventure 
story of the television show, and a "Maya Calendar" 
used to explore how the Maya kept track of time by 



Figure 1. Counting to'.2l in the Maya rmmberi^rstem 




Each episode of the drama is followed by a docu- 
mentary expedition, in which one of the young actors 
visits a scientist and explores in depth a topic touched 
upon in the dramatic series. The computer classroom 
modules are also used to further elaborate upon themes 
contained in the videos. In addition to the Maya Math 
program, the second season classroom modules in- 
clude a solar astronomy simulation. 

The Maya Math program is designed to present a 
particular aspect of mathematics as an archaeological 
discovery. In the module, students are asked to adept 
the role of archaeologists attempting to decipher the 
mearing of Maya number glyphs. (The Maya number 
system is a base twenty system that uses three number 
symbols: a shell stands for zero, a dot stands for one, 
and a bar stands for five. See Figure 1.) 
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Figure 2. Maya Calculator 




counting days. 

The formative research carried out in conjunction 
with the design process was conducted over a four- 
month period and involved both the software and 
printmaterials.Fivefourth- through sixth-gradeclass- 
rocms were involved in the evaluation. The students 
came from a variety of socioeconomic groups, and the 
teaching styles varied from more traditional teacher- 
led approaches to student-centered classrooms. 

The research was conducted in three pSases: The 
first involved testing the concepts central to the work- 
ings of Maya Math. The second phase involved evalu- 
atinga prototype version of the Maya Calculator, and 
the third involved evaluating the program as a whole 
in conjunction with a draft of the print materials. 
Rather than concentrating on each phase of research 
as a distinct entity, this paper will discuss the three 
factors that must be assessed in the development of 
anv educational software program: comprehensibil- 
ity, appeal, and usefulness. 



The Assessment of Comprehensibility 

In many respects, assessing the comprehensibility of 
educational software is the most ditficult and also the 
most impx:>rtant task the formative researcher faces. 
On thesuridce, comprehensibility research appears to 
be a straightforward investigation of the clarity of the 
educational ideas contained within the software and 
the clarity of the user interface. The researcher, how- 
ever, soon runs up against an inevitable fact — stu- 
dents of differing abilities must be able to use the 
software effectively. Part of wha*- is involved in as- 
sessing comprehensibility is determining the range of 
variability instudents'understandingof the software's 
central concepts and making recommendations to the 
design team about ways in which to accommodate 
various abilities. In addition, an assessment of com- 
prehensibility is necessary to createabasisfrom which 
the issues of appeal and usefulness can be interpreted. 
Comprehensibility research, therefore, lays an impor- 
tant foundation from which other issues that are of 
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concern to the design team can be investigated. 

During the earliest phases of our research, we 
designed an off-line version of the Maya Calculator 
using a cardboard stela (an upright stone slab carved 
with inscriptions, on which the Maya typically carved 
historical information) and paper cutouts of dots, 
bars, and shells. Our f urpose in doing this was two- 
fold: We wanted to get an indication of the kinds of 
conceptual problems students were likely to encoun- 
ter as they worked with different number symbols in 
another place value system, and we wanted to assess 
the range of student understanding. We found that all 
the students we worked with were quite capable of 
using the Maya number syn^bols to build numbers 
less than twenty. However, it became clear that stu- 
dents were at various levels in their understanding of 
the concept of place value when they began to work 
with these number symbols in a base twenty system 

The students fell into th) :e easily identifiable 
groups. The first was composed of students who 



appeared to havea tecnnical or rigid understanding of 
phce value. Given an Arabic number, they were able 
to identify the one's, ten*s, hundred's, and thousand's 
place, but they had no underlying conceptual basis for 
understanding how place value works. Although these 
students were able to convert Maya numbers less than 
nineteen, they were at a complete loss when asked to 
convert Maya numbers greater than twenty. The most 
common error they made was to ignore the operations 
of place and add all the Maya dots and bars together. 
For example, when given a number that had a bar in 
the one's place and a dot in the twenty's place, the 
students would decode this number as six instead of 
twenty-five (see Figure 3). 

For students who lack a conceptual framework in 
which to locate Maya Math, the arithmetical opera- 
tions involved in this program prove to be mysteri- 
ous. Not only do students need to remember what the 
Maya symbols st?nd for and the value of ihe different 
places, but they also have to apply the correct algo- 



Figufe 3. Incoh^egtdamf correcKprocedures for decoding the Maya Rumfeer 25 
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rithm to decode or encode a given number. Once the 
design team had infornnation about the specific areas 
in which these students were havir^ difficulty, they 
were able to make changes in the software that accom- 
modated the particular needs of this group. We de- 
cided that these students needed a simple but moti- 
vating route into working with Maya numbers, which 
led eventually to the creation of Glyph Trek, a number 
dw.oding game that can be played at a very elemen- 
tary level. 

Additional research made it apparent that our 
prototype help screen did not aid these students in 
distinguishing among the various operations that are 
necessary when translating Maya numbers into their 
Arabic counterparts. We agreed that a traveling cur- 
sor should be used to selectively highlight each algo 
nihm in order to differentiate one operation from the 
next. Finally, in the accompanying print materials, a 
number of basic, ye^ engaging activities were de- 
signed to help this group develop a better under- 
standing of place value. 

The second group of students had a much easier 
time with the concept of place value than did the first 
group. Although, like the first group, they lacked a 
solid conceptual understanding of place value, with 
practice they developed a firmer grasp of the concept. 
The design changes made to accommodate the first 
group also proved to be extremely helpful to these 
students. We decided to incorporate different levels 
into Glyph Trek so that these students could play a 
more challenging game while developing their un- 
derstanding of the basic, yet essential operations in- 
volved in Maya math. Using the traveling cursor io 
highlight the help screen also proved to be useful in 
alleviating misunderstandings. 

These students presented anadditional challenge. 
Once they had mastered some of the fundamental 
concepts, we were faced with the "so what's next" 
problem. Many students assumed that they had 
learned all that there was to learn about Maya Math, 
when they were no t in fac t ready to take on some of the 
more challenging aspects of the software. Their inabil- 
ity to meet certain challenges successfully was a major 
source of discouragement. 

We were faced with the problem of creating an 
intermediate activity that would hold students' inter- 
est while at the same time helping them to develop a 
solid understanding of place value concepts. We c**e- 
ated an off-line version of the game "21/ called 'To 



the Limit," in which students, playing in pairs, roll 
either Maya or Arabic number dice and attempt to be 
the first to reach 21. In this game, the player may 
decide to subtract her/his roll from the opponent's 
total or add to her/his own total. Tliis activity in- 
volves all of the essential elements of doing arithmetic 
using Maya numbers — grouping, trading, and place 
value. When students have mastered this simple ver- 
sion, they can choose a larger number as their goal. 
This game was a huge success, serving as a kind of 
transitional environment that enabled students to 
move on to more complicated aspects of the software. 

The third group of students we encountered had 
a well-developed conceptual understanding of the 
concept of place value. They had little trouble, if any, 
in mastering the Maya number ^yslem. The problem 
that we faced was how to ensure that the software 
would be rich enough in challenges to hold their at- 
tention for sustained periods of time. These students 
often began playing Glyph Trek at the hardest level 
and quickly lost interest. Similarly, they often lelt K\at 
they had nothing to learn from the calculator and were 
easily bored, '^e formative research n\^de it very 
clear that the software and print materials would have 
to include a number of stimulating activities aimed 
directly at this group of ''tudents. As a result, the 
calculator contains a number of features that can be 
used as challenge activities; for example, the com- 
puter picks a base and the student guesses what it is, 
or the student hides the base and challenges a partner 
to guebs it. In addition, the print materials contain 
mystery number puzzles as well as a number of 
intriguing thought problems that require the help of 
the calculator to solve. The print naterials also en- 
courage these more advanced stuaents to play To the 
Limit in different number bases. 

The Assessmenf of Appeal 

Although not as difficult to assess as comprehensibil- 
ity, the evaluation of a software program's appeal 
contains its own set of problems. Most important, it is 
necessary to ascertain whether a student genuinely 
dislikes a program or whether she/he dislikes the 
program because of a failure to understand some 
cntical aspect of it. In the development of Maya Math 
we were always walking a fine line discriminating 
between these two phenomena. 

The calculator, in particular, presented us with an 
interesting challenge. At the outset of the develop- 
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ment process, the design team made a conscious 
decision to create a software tool rather than a soft- 
v/are game. There were several reasons for this, rang- 
ing fron* the 'nitial scope of the project, to the amount 
of time available, to the fact that thesubject matter lent 
itself nicely to the Iculator format. The most imme- 
diate reaction we received from students was cne of 
displeasure. Some of this displeasure was genuine 
and some, as should be clear from the above discus- 
sion, had to do with the fact that certain students were 
unable to understand basic components of the soft- 
ware program. 

With the first group of students, it was apparent 
that their displeasure had to do with their lack of 
understanding. It became clear that they needed an 
entertaining and motivating path into the calculator. 
The result, as explained above, was Glyph Trek. 

The other two groups of students presented us 
with a different dilen^ma. Although they were ini- 
tially interested in and curious about the calculator, 
their interest waned with time. The research we car- 
ried out made it apparent that the calculator was 
simply not entertaining enough. Our recommenda- 
tion to the design team was to createan on-line version 
of the To the Limit game. Time and resources inter- 
vened, however, and the game never materialized as 
an on-line activity. As an alternative, the print materi- 
als contain a note to teachers suggesting that students 
create their own game boards and play off-line. The 
reality of any software development project is that 
there are always instances in which priorities other 
than research must prevail. More, often than not, 
decisions come down to what is economically fea- 
sible. 

The Assessment of Usefulness 

Finally, determining the educational effectiveness of 
a software product is something that formative re- 
search can only evaluate in the most superficial way. 
To assess whether or not a program actually helps a 
child learn something of value is the prerogative of 
more formally planned academic research. However, 
the formative researcher can get a general reading on 
the educational effectiveness of the product and she/ 
he can also assess whether the product is useful to 
teachers. 

The formative research carried out on Maya Math 
undertook to do two things: 



1. From the beginning of the development proc- 
ess, W2 invited teachers to comment on the educa- 
tional objectives of the program, the design of the 
software, and the content and form of the print mate- 
rials. 

2. We also paid close attention to students' initial 
imderstanding of the software and whether their 
understanding seemed to improve as they worked 
with the program. 

Obtaining teachers' input is essential to thedevel- 
opment of any software program that will be used 
within school settings. We organized a series of panels 
for teachers to view a prototype version of Maya Math 
and h3ar a presentation on the educational objectives 
of the program. The suggestions that teachers made 
were extremely useful in helping us to refine the 
product and place its educational goals in a context 
that would speak clearly to teacher needs. Teachers' 
comments on the importance of understanding place 
value by working with other number systems sup- 
ported our decision to make the calculator function in 
any base between two and twenty. 

Four classroom teachers were involved in an 
extensive field test, in which tho final version of the 
software was evaluated in conjunction with the print 
materials. We were fortunate to have the time avail- 
able to assess how these materials worked on an 
ongoing basis within several different classroom*;. As 
a result, many important modifications were made. 
For example, it became apparent that the best route 
into understanding the Maya number system was 
through a comparison with our own base ten number 
system. Several of the teachers with whom we worked 
approached the materials with this in mind. They 
created charts that illustrated the similarities and 
differences between the two number systems and 
they designed lessons in which their students were 
asked to make important comparisons. Many of their 
final suggestions were incorporated into the print 
materials. 

As mentioned above, without undertaking a for- 
mal study it is difficult to know whether what stu- 
dents learn can be attributed to the softwarematerials. 
In any evaluative study there are always numerous 
variables to untangle; teaching style, the amount of 
time students spend using the software, the specific 
approach students and teachers take, as well as the 
students' previous knowledge bases can all affect 
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learning outcomes. The formative researcher, how- 
ever, can use information gathered in the evaluation 
of comprehensibility to form an impression about 
what students are learning. 

The three groups of users that our research iden- 
tified helped us to evaluate whether students' under- 
standing of the materials improved over time. Not 
surprisingly, students in the middle group seemed to 
develop a better understanding of place value than 
did students in the other two groups. It is conceivable 
that students in this group had enough of a knowl- 
edge base to build upon successfully. Students in the 
first group appeared to be hindered by the fact that 
their understanding of place value was principally 
algorithmic. As a result, they lacked the conceptual 
flexibility that is necessary to be a successful user of 
Maya Math. In contrast, students in the third group, 
who did have a well-developed conceptual under- 
stand i ng of place value, may have reached a plateau i n 
their capacity to learn more about this particular 
concept. 

In conclusion, I would like to call attention to the 
fact that this discussion of formative research makes it 
appear to be a more logical and procedural process 
than in fact it is. In actuality, formative research is 
highly intuitive and is often learned in Ihe course of 
carrying out the work. The researcher must possess 



certain qualities that take time and practice to de- 
velop. The ability to engage with students and listen 
to their concerns, while at the same interpreting with 
a critical eye, is central to the process. In addition, the 
capacity to work with diverse groups of people, such 
as designers, educators, psychologists, and adminis- 
trators, and juggle what areoften conflicting demands 
is essential. In the final analysis, however, the forma- 
tive researcher represents the audience who will use 
the materials. In this case, we served as the students' 
advocate, ensuring that the process of learning could 
be challenging, successful, and fun. 
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